Obstacle detection device and method therefor

ABSTRACT

Provided is a technology for helping safe driving and realizing automatic driving of vehicles, or for counting the number of passing vehicles on the road or monitoring those passing vehicles for their driving. Using a plurality of cameras mounted in a vehicle or provided above a road, even if the relationship between the road plane and the respective cameras constantly changes in relative position or posture due to camera vibration or a change in road tilt, any obstacles located on the road such as other vehicles ahead, parked vehicles, and pedestrians on the road are detected without confusing those with textures including white lines, road signs, paint, road stains, and shadows of roadside objects, all of which do not disturb vehicle driving. An obstacle detection device  10  is structured by an image input section  11  for receiving images from a plurality of image pick-up devices  101 , a correspondence detection section  12  for finding a plurality of pairs of corresponding points from the received right and left images, the slope degree calculation section  13  for calculating a slope degree of a plane including the corresponding points, and a result determination section  14  for determining as there being an obstacle when the calculated slope degree is larger than a predetermined value.

This is a continuation of Application No. 10/649,939, filed Aug. 28,2003, now U.S. Pat. No. 6,906,620 which is incorporated herein byreference.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application is based upon and claims the benefit of priority fromthe prior Japanese Patent Application No. 2002-249782, filed on 28 Aug.2002; the entire contents of which are incorporated herein by reference.

BACKGROUND OF THE INVENTION

1. Field of the Invention

The present invention relates to an obstacle detection device fordetecting any obstacles on a road without confusing those with texturesusing a plurality of cameras mounted in a vehicle or provided above theroad, and a method applied to the device. Here, the obstacles are thoselocated on the road such as other vehicles ahead, parked vehicles, andpedestrians. The textures include white lines, road signs, paint, roadstains, and shadows of roadside objects, all of which do not disturbvehicle driving. The cameras are provided mainly for helping safedriving and realizing automatic driving of the vehicle, or for countingthe number of passing vehicles on the road or monitoring those passingvehicles for their driving. Obstacle detection is possible even if therelationship between the road plane and the respective camerasconstantly changes in relative position or posture due to cameravibration or a change in road tilt, for example.

2. Description of the Background Art

A method for detecting any obstacles on roads is classified into twotypes: one is a type using active sensors typified by sonars, radars,and range finders; and the other is a type using passive sensorstypified by visible-light CCD cameras and infrared cameras.

The active sensors are popular for measuring object positions in variousapplications, and well known for their usability. The active sensorshave, however, problems for an application of detecting any obstacleslying in the vehicles' way on the roads such as other vehicles.Specifically, the problems are associated with low detection resolution,not enough measurement range, erroneous detection of non-obstacles onthe roads, and erroneous detection of objects lying on non-disturbingroad side due to no driving lane detection capability. Thus, there hasbeen a demand for an advanced obstacle detection technology by imageanalysis using the passive sensors exemplified by CCD cameras.

To detect obstacles lying on road surfaces through analysis of imagesprovided exemplarily by CCD cameras mounted in vehicles, generallyutilized is information about image brightness intensity pattern ordriving lanes recognized for the purpose. As for detecting the drivinglanes, cutting out parts in shades of gray with less texture will dofrom images picked up by a camera.

The issue here is that, many obstacles are actually similar to roads inbrightness intensity or pattern, resulting in difficulty achieving thehigher usability with the less erroneous detection.

There is another type of method using a plurality of cameras fordetecting obstacles and driving lanes. Such a method is generally calleda stereoscopic method.

With stereoscopic views, three-dimensional (3D) information about atarget detection region can be derived on the triangulation principle.Thus, stereoscopic views seem to be a solution for obstacle and lanedetection with higher accuracy, but still bear problems. For example, acorresponding point search cannot be uniquely solved, and thecalculation cost is quite expensive. This corresponding point search isdone to find any specific point(s) in real world shared by a pluralityof camera images.

In this respect, methods disclosed in Patent Literature 1(JP-A-2000-293693) and Patent Literature 2 (JP-A-2001-76128) do notrequire such a corresponding point search, and are considered useful forobstacle detection. These methods are described in the below.

Assuming now that two cameras, right and left, are provided to pick upimages of a road. Project points as a result of projecting points on theroad plane onto images picked up by the right and left cameras arepresumably (u, v) and (u′, v′), and a relational expression 1 isestablished as follows:

$\begin{matrix}{{u^{\prime} = \frac{{h_{11}u} + {h_{12}v} + h_{13}}{{h_{31}u} + {h_{32}v} + h_{33}}},\mspace{31mu}{v^{\prime} = \frac{{h_{21}u} + {h_{22}v} + h_{23}}{{h_{31}u} + {h_{32}v} + h_{33}}}} & (1) \\{h = \left( {h_{11},h_{12},h_{13},h_{21},h_{22},h_{23},h_{31},h_{32},h_{33}} \right)} & (2)\end{matrix}$

The equation 2 shows a parameter dependent on the positions and posturesof the right and left cameras with respect to the road plane, lens focaldistances of the cameras, points of origin of the images, and the like.The parameter h can be derived in advance only by project points (u_(i),v_(i)) and (u_(i)′, v_(i)′) (i=1, 2, . . . , N), which are those derivedby projecting four or more points on the road plane onto the right andleft images. Using such a relational expression, a corresponding pointP′(u′, v′) on the left image is derived based on the assumption that anarbitrary point P(u, v) on the right image is located on the road plane.If the point P is truly located on the road plane, the points P and P′are paired as the correct corresponding points, leading to a good matchbetween two pixels or neighboring regions in terms of brightnessintensity or feature. On the other hand, if the points P and P′ differin brightness intensity, the point P is determined as belonging to anobstacle region. This method allows for determining whether an arbitrarypoint in the image has a height from the road plane directly only fromthe relational expression 1. There is thus no need for the correspondingpoint search between the right and left images.

To apply such a scheme for obstacle detection in front of the vehicle,the parameter h is presumed as roughly constant when the vehicle isdriving on rather flat road at low speed. Thus, there is no need tocalculate the parameter h twice for correct obstacle detection.

Here, as to correspondence detection, the operation of a sectionprovided therefor is described by referring to FIGS. 3 to 5.

The correspondence detection section operates to convert a first imagepicked up by a first image pick-up device into an image viewed from aviewpoint of a second image pick-up device. A parameter used for thisconversion is so calculated as to keep a typical geometric relationshipbetween a plurality of image pick-up devices and the road plane, with apresumption that the vehicle is standing still on the no-tilting roadplane. The parameter is not calculated twice, and not changed duringobstacle detection, e.g., when the vehicle is moving.

The parameter is calculated in a manner based on the Patent Literature1, and described in the below.

Referring to FIG. 3, two cameras a and b are set up. The road surfacehas two parallel white lines l and l′ extending roughly along theoptical axes of the cameras. The obstacle detection device is notnotified of the relationship between the two cameras a and b in positionand posture, but only of epipolar constraint. During when the obstacledetection device is in operation, no change occurs, presumably, to therelative positions and postures of the cameras a and b, and epipolarconstraint. Here, the epipolar constraint means a constraint conditionfor stereoscopic images of a general type. Under this condition, asshown in FIG. 4, the arbitrary point P on the image (right image) pickedup by the camera a is so constrained as to be on a predetermined linearline including the corresponding point P′ on the image (left image)picked up by the camera b. This linear line is referred to as anepipolar line. As an example, when the optical axes of the cameras areso placed as to be parallel to each other, the corresponding point ofthe arbitrary point P in the right image is found on the same scanningline on the right image. Accordingly, the epipolar line agrees with thescanning line. The epipolar constraint is dependent on the relationshipbetween the stereoscopic cameras in relative position and posture, andinternal parameters of the cameras, e.g., lens focal distance, originpoint of images. Thus, the epipolar constraint being invariant means therelative positional relationship between the stereoscopic cameras andtheir internal parameters showing no change (during when the obstacledetection device is in operation or the vehicle having the devicemounted therein is moving). This epipolar constraint is formulated asthe following equation 3.(u,v,1)F(u′,v′,1)^(T)=0  (3)

Herein, (u, v) is the arbitrary point P on the right image, and (u′, v′)is the corresponding point of the point P on the left image. F denotes a3×3 matrix, and referred to as Fundamental matrix. Expanding theequation 3 will lead to the following equation 4.(F ₁₁ u+F ₁₂ v+F ₁₃)u+(F ₂₁ u+F ₂₂ v+F ₂₃)v′+(F ₃₁ u+F ₃₂ v+F₃₃)=0  (4)

Herein, F_(ji) (i, j=1, 2, 3) denotes an element of j row(s) and icolumn(s) of the matrix F, and can be derived from a plurality ofcorresponding points. Further, the equation 4 denotes an epipolar linecorresponding to the point P(u, v) on the right image. Nine elements ofthe matrix F are not all independent, and theoretically, are derivablefrom seven corresponding points. Because 3D position is not required foreach pair of the corresponding points, calculating the matrix F, i.e.,the epipolar constraint, is rather easy. The lines l and l′ in eachimage are parallel three-dimensionally but not on the images picked upby the right and left cameras. As shown in FIG. 5, the lines l and l′ ineach image cross each other at a point at infinity, which is called avanishing point. Next, derived is a relationship established between thecorresponding points on the road plane. As shown in the right image ofFIG. 5, arbitrary points on the linear line l are P₁ and P₃, andarbitrary points on the linear line l′ are P₂ and P₄. For these fourpoints, corresponding points P₁′, P₂′, P₃′, and P₄′ in the left imagecan be calculated using the epipolar constraint previously derived. Thatis, the point P₁′ correspond to the point P₁ agrees with an intersectionpoint of the linear line l and the epipolar line L₁ of the point P₁ onthe left image. Similarly, the points P₂′, P₃′, and P₄′ can be derivedas intersections, respectively, of the epipolar lines L₂, L₃, and L₄ ofthe points P₂, P₃, and P₄, and the linear line l or l′. Assuming thatcoordinates of the point P_(i) (i=1, 2, 3, 4) are (u_(i), v_(i)), andcoordinates of the point Pi′ (i=1, 2, 3, 4) are (u_(i)′, v_(i)′). Therelation between the coordinates (u_(i), v_(i)) and (u_(i)′, v_(i)′) canbe expressed by a relational expression 5.

$\begin{matrix}{{{u_{i}^{\prime} = \frac{{h_{11}u_{i}} + {h_{12}v_{i}} + h_{13}}{{h_{31}u_{i}} + {h_{32}v_{i}} + h_{33}}},\mspace{31mu}{v_{i}^{\prime} = \frac{{h_{21}u_{i}} + {h_{22}v_{i}} + h_{23}}{{h_{31}u_{i}} + {h_{32}v_{i}} + h_{33}}}}\left( {{i = 1},2,3,4} \right)} & (5)\end{matrix}$

These eight equations are solved using the following equation 6.h=(h₁₁,h₁₂,h₁₃,h₂₁,h₂₂,h₂₃,h₃₁,h₃₂,h₃₃)  (6)

If an arbitrary solution h satisfies the equation 5, a constant multiplekh of h (k is constant) also satisfies the equation 5. No generality isthus lost with h₃₃=1, and eight equations will lead to h composed ofnine elements. By using such derived h, the corresponding point P′(u′,v′) on the right image can be calculated as the following equation 7with an assumption that the arbitrary point P(u, v) on the left image islocated on the road plane.

$\begin{matrix}{{u^{\prime} = \frac{{h_{11}u} + {h_{12}v} + h_{13}}{{h_{13}u} + {h_{32}v} + h_{33}}},\mspace{31mu}{v^{\prime} = \frac{{h_{21}u} + {h_{22}v} + h_{23}}{{h_{31}u} + {h_{32}v} + h_{33}}}} & (7)\end{matrix}$

With the methods in Patent Literatures 1 and 2, when the vehicle driveson typical outside roads, the relationship between the road plane andthe respective cameras continuously changes in relative position andposture due to vibrations occurring to the obstacle detection device, ora change in road tilt. Consequently, these methods bear such problems,due to vehicle vibration, as frequent erroneous detection especiallyaround the texture on the road plane such as white lines, road signs,paint, road stains, shadows of roadside objects and vehicles, and thelike.

As described above, with an obstacle detection device using CCD camerasof a conventional type, usage environment is limited, or therelationship between the road plane and the respective camerascontinuously changes in relative position and posture due to vibrationsduring the device operation or driving vehicle. As a result, frequenterroneous detection occurs especially around the texture on the roadplane such as white lines, road signs, paint, road stains, shadows, andthe like, considerably lowering the true detection accuracy of obstacledetection.

The present invention is proposed in consideration of the aboveconventional problems, and an object thereof is to provide an obstacledetection device capable of correctly detecting only true obstacles nomatter what road the device is set up, or no matter what road a vehiclehaving the device mounted therein is driving.

SUMMARY OF THE INVENTION

An embodiment of the present invention is an obstacle detection devicein which at least two image pick-up devices each pick up an image of apreset common detection region, and which determines whether an obstacleis present or not in the detection region from stereo images picked upusing the image pick-up devices. The device includes: image input meansfor receiving the image from each of the image pick-up devices;correspondence detection means for deriving a plurality of pairs ofcorresponding points in the detection regions of the received two stereoimages; slope degree calculation means for calculating a slope degreecorresponding to a slope angle between a basic plane which is parallelto optical axes of the image pick-up devices and a detection planeincluding the derived plurality of corresponding points in the stereoimages; and result determination means for determining as there being anobstacle when the slope degree of the detection plane is larger than apredetermined value.

According to a second aspect, in the first aspect, the slope degreecalculation means regards the slope angle as a pitch angle between thebasic plane and the detection plane, and a parameter of an affinetransformation matrix indicating a relationship between thecorresponding points in the images or a unique value derived from theparameter as a slope degree.

According to a third aspect, in the first aspect, the slope degreecalculation means calculates the slope degree by solving an equationbased on a parallax and vertical positions of the plurality of pairs ofcorresponding points.

According to a fourth aspect, in the first aspect, the slope degreecalculation means calculates the slope degree by voting coordinatevalues of the plurality of pairs of corresponding points into a votingspace based on an equation satisfied by the coordinate values.

According to a fifth aspect, in the first aspect, the slope degreecalculation means detects, as an obstacle, out of the plurality of pairsof corresponding points, only the pair of corresponding points resultingin the slope degree large in value.

An embodiment of the present invention is directed to an obstacledetection method in which at least two image pick-up devices each pickupan image of a preset common detection region, and which determineswhether an obstacle is present or not in the detection region fromstereo images picked up using the image pick-up device. The methodincludes: an image receiving step of receiving the image from each ofthe image pick-up devices; a correlation detecting step of deriving aplurality of pairs of corresponding points in the detection regions ofthe two stereo images; a slope degree calculating step of calculating aslope degree corresponding to a slope angle between a basic plane whichin parallel to optical axes of the image pick-up devices and a detectionplane including the derived plurality of corresponding points in thestereo images; and a result determining step of determining as therebeing an obstacle when the slope degree of the detection plane is largerthan a predetermined value.

An embodiment of the present invention is a program for realizing, bycomputer execution, an obstacle detection method in which at least twoimage pick-up devices each pick up an image of a preset common detectionregion, and which determines whether an obstacle is present or not inthe detection region from stereo images picked up using the imagepick-up device. The program realizes: an image input function forreceiving the image from each of the image pick-up devices; acorrespondence detection function for deriving a plurality of pairs ofcorresponding points in the detection regions of the two stereo images;a slope degree calculation function for calculating a slope degreecorresponding to a slope angle between a basic plane which is parallelto both of optical axes of the image pick-up devices and a detectionplane including the derived plurality of corresponding points in thestereo images; and a result determination function for determining asthere being an obstacle when the slope degree of the detection plane islarger than a predetermined value.

An embodiment of the present invention is an obstacle detection devicein which at least two image pick-up devices each pick up an image of apreset common detection region, and a determination is made from stereoimages picked up using the image pick-up devices whether or not thedetection region includes an obstacle. In the device, included are: animage input section for receiving the stereo images from the imagepick-up devices; a correspondence detection section for deriving aplurality of pairs of corresponding points in the detection regions ofthe received stereo images; a slope degree calculation section forcalculating a slope degree corresponding to a slope angle between abasic plane which is parallel to optical axes of the image pick-updevices and a detection plane including the derived plurality ofcorresponding points of the stereo images; and a result determinationsection for determining as there being an obstacle when the slope degreeof the detection plane is larger than a predetermined value.

An embodiment of the present invention is a recording medium havingrecorded a program for realizing, by computer execution, an obstacledetection method in which at least two image pick-up devices each pickup an image of a preset common detection region, and a determination ismade from stereo images picked up using the image pick-up deviceswhether or not the detection region includes an obstacle. In the method,realized are: an image input function for receiving the stereo imagesfrom the image pick-up devices; a correspondence detection function forderiving a plurality of pairs of corresponding points in the detectionregions of the received stereo images; a slope degree calculationfunction for calculating a slope degree corresponding to a slope anglebetween a basic plane which is parallel to optical axes of the imagepick-up devices and a detection plane including the derived plurality ofcorresponding points in the stereo images; and a result determinationfunction for determining as there being an obstacle when the slopedegree of the detection plane is larger than a predetermined value.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic diagram showing the structure of an obstacledetection device according to an embodiment of the present invention;

FIG. 2 is a schematic diagram showing the set-up state of image pick-updevices mounted in a vehicle;

FIG. 3 is a schematic diagram showing a road plane and the set-up stateof image pick-up devices;

FIG. 4 is a schematic diagram showing epipolar constraint applied to anarbitrary point;

FIG. 5 is a schematic diagram showing epipolar constraint applied pointson white lines l and l′;

FIG. 6 is a schematic diagram illustrating a detection region image withno obstacle;

FIG. 7 is a schematic diagram illustrating a detection region image withany obstacles;

FIG. 8 is a schematic diagram showing a parallax on a line segment in adetection region;

FIG. 9 is a schematic diagram showing a case where a slope is observedahead of cameras;

FIG. 10 is a schematic diagram showing image conversion in a case wherea road sign is placed on the slope;

FIG. 11 is a schematic diagram showing a method for calculating slopedegrees;

FIG. 12 is a schematic diagram showing a voting method at the time ofslope degree calculation;

FIG. 13 is a schematic diagram showing pairs of corresponding points ina case where a detection region or any corresponding point is displaced;and

FIG. 14 is a schematic diagram showing a slope degree calculation in acase where a detection region or any corresponding point is displaced.

DETAILED DESCRIPTION

In the below, an embodiment of the present invention is described byreferring to FIGS. 1, 2, and 6 to 14.

FIG. 1 is a schematic diagram showing the structure of an obstacledetection device 10 of the present embodiment.

The obstacle detection device 10 is structured by an image input section11, a correspondence detection section 12, a slope degree calculationsection 13, and a result determination section 14.

When the result determination section 14 determines as there being anobstacle, a warning device 102 may warn a driver or a supervisor.Further, if the obstacle detection device 10 is mounted in a vehicle,vehicle body control means 103 may be provided for braking.Alternatively, vehicle braking by steering is a possibility. In anycase, a communications device 104, or the like, may be provided toreceive any output or transferred result.

In the obstacle detection device 10, the image input section 11, thecorrespondence detection section 12, the slope degree calculationsection 13, and the result determination section 14 are functionallyrealized by computer-stored programs.

1. Image Input Section 11

The image input section 11 receives images from arbitrary number ofimage pick-up devices 101. In the present embodiment, the image pick-updevice 101 exemplified by a CCD camera are plurally provided. In a casewhere two of the devices are mounted in a vehicle, as shown in FIG. 2,those are attached to the front part of the vehicle, on the right andleft sides. If two are provided above a road, as shown in FIG. 3, thoseare so provided as to face in the direction the road is extending. Theoptical axes of these two image pick-up devices 101 are parallel to eachother.

The image input section 11 subjects, to A/D conversion, video signalscoming from a plurality of image pick-up devices. Then, the A/Dconversion result is stored in memory of the image input section 11 asdigital images or image sequences. In response to a request, the imagesof arbitrary time and arbitrary region are output to the correspondencedetection section 12 in the following stage.

2. Correspondence Detection Section 12

The correspondence detection section 12 performs corresponding pointsearch with respect to the images picked up by the image pick-up devices101 and stored in the image input section 11.

Specifically, the corresponding point search is done to find anyspecific point(s) in real world shared by the images picked up by thecameras. In this embodiment, thus found corresponding points are usedfor calculating a slope degree in the slope degree calculation section13 in the next stage. Here, 3D structure reconstruction is not done bythe general stereoscopic image processing. Thus, the corresponding pointsearch is not necessarily strictly done, and any simpler methodrequiring less calculation can be used.

Considered now is a case where a corresponding points are derivedbetween an image picked up by the right camera and stored in the imageinput section 11 (hereinafter, referred to as right image) and an imagepicked up by the left camera at the same time (hereinafter, left image).

2-1. Calculation of Feature Quantity of Right Image

First, arbitrary pixels or arbitrary regions in the right image arecalculated for their predetermined feature quantities. Any pixel orregion large in feature quantity is selected as a feature point of theright image.

Herein, the “feature quantity” denotes arbitrary information derivablefor each pixel or in an arbitrary region. Possibly used as the featurequantity are simply a pixel brightness intensity value or colorinformation, or an arbitrary scalar or vector calculatable from resultsderived by arbitrary object detection or a region segmentation processsuch as differentiation or integration result in space or time,arbitrary filter superposition result, or mean or variance statistic, orcalculatable from images such as feature quantities or movementquantities of a region derived by these results. In this example,described is a case where the feature quantity is a brightness intensitypitch at a pixel position.

Here, the “intensity gradient” denotes a degree of brightness intensitychange in the vicinity of the pixels, and the closer to the boundaryregions, i.e., edges, of objects or object structures in images, thelarger the value gets.

To calculate such a brightness intensity pitch, a filter such as a Sobeloperator may be applied to a target image.

If the Sobel operator is used simply for deriving a brightness intensitypitch in the vertical direction of the image, the following equation 8can be used.

$\begin{matrix}{{{D\left( {x,y} \right)} = {\sum\limits_{i,{j = {{- {M\ldots}}\mspace{11mu} M}}}{{{sgn}(i)}{I\left( {{x + i},{y + j}} \right)}}}}{{{sgn}(i)} = \left\{ \begin{matrix}{- 1} & \left( {i < 0} \right) \\0 & \left( {i = 0} \right) \\{1} & \left( {i > 0} \right)\end{matrix} \right.}} & (8)\end{matrix}$Herein, a brightness intensity value of a pixel (x, y) is I(x, y), D(x,y) is a brightness intensity pitch value in the vertical direction, and2M+1 is the filter size. From the pixel (x, y) whose absolute value|D(x, y)| being large, an arbitrary number of pixels are regarded as thefeature points in the right image. Alternatively, as for a certainthreshold value Th, any pixel (x, y) satisfying |D(x, y)|>Th are thefeatures points in the right image.

Various other filters can be used for deriving a brightness intensitypitch, including Laplacian, Robinson, and Canny, and any will do. Otherthan a well-known method for deriving a brightness intensity pitch,there are many methods for calculating the feature quantity of images.The details are described in, for example, Non-patent Literature 1(edited by Takagi and Shimoda, Image Analysis Handbook, Tokyo UniversityPress, ISBN4-13-061107-0).

2-2. Detection Corresponding Point in Left Image

Next, calculated is a point in the left image correspond to the featurepoint in the right image.

Methods for calculating a matching level between arbitrary pixels orarbitrary regions is called matching, and typified by template matching.With the template matching, a region in the vicinity of the featurepoint in the image is used as a template, and the template is comparedto an arbitrary part of the target search region in the image to find aregion showing a tight match.

2-2-1. Method Using SAD

In a method using the easiest SAD (Sum of Absolute Difference), thefollowing equation 9 is solved for each pixel (x, y) in the searchregion. The pixel (x, y) resulting in the smallest R(x, y) is derived asa corresponding point. Herein, a template region is K, a brightnessintensity value of a template pixel is T(i, j) (in this embodiment, theregion in the vicinity of the feature point derived for the rightimage), and a brightness intensity value of the pixel in the searchregion is I(x, y) (in this embodiment, the search region set for thefeature point in the left image).

$\begin{matrix}{{R\left( {x,y} \right)} = {\sum\limits_{{({i,j})} \in K}{{{T\left( {i,j} \right)} - {I\left( {{x + i},{y + j}} \right)}}}}} & (9)\end{matrix}$Here, as described referring to FIG. 4, the corresponding point islocated on an epipolar line. Thus, the search region of thecorresponding point is limited onto the epipolar line, thereby reducingcalculating task for the corresponding point search. Further, in a casewhere the optical axes of the cameras are parallel to each other, theepipolar line agrees with the scanning line, making the correspondingpoint search easier.2-2-2. Other Methods

There are other various methods for deriving corresponding points,including a method using SSD (Sum of Squared Difference), a sequentialsimilarity detection algorithm (SSDA), a normalized correlationcoefficient method, structure matching, and the like, and any arbitrarymethod is applicable. The above Non-Patent Reference 1 shows detailsabout well-known methods.

2-2-3. Case of Calculating a Plurality of Corresponding Points for OneFeature Point

Alternatively, a plurality of corresponding points in the left image maybe searched for one feature point in the right image.

In the above, after the corresponding point search by template matchingusing SAD, only a point (x, y) resulting in the smallest R(x, y) isregarded as a corresponding point. Here, on the other hand, regarded ascorresponding points are points (x′, y′) and (x″, y″) resulting in,respectively, the second and third smallest R(x′, y′) and R(x″, y″), forexample. In this manner, an arbitrary number of corresponding points inthe left image can be derived for one feature point in the right image.

2-2-4. Other Corresponding Point Search

Such a corresponding point search is applicable not only to originalimages, i.e., brightness intensity value images, but also to images as aresult of the arbitrary feature quantity calculation described in theabove.

3. Slope Degree Calculation Section 13

The slope degree calculation section 13 uses pairs of correspondingpoints in the right and left images derived by the correspondencedetection section 12 to find a slope degree corresponding to a slopeangle of a plane in a 3D space in the detection region (in the below,referred to as detection plane).

Here, the “slope angle” is an angle between the road plane and thedetection plane, and is a pitch angle with respect to the road plane,having small yaw and roll angles. Details are left for laterdescription. Note that, the road plane is a plane (basic plane) which isparallel to optical axes of two image pick-up devices 101.

The slope angle is not a tilt to a horizontal plane but a tilt to theroad plane. Even if a vehicle having the image pick-up device 101mounted therein is driving on a road plane tilted against the horizontalplane, i.e., slope, the “slope angle” between the road plane and a planeparallel thereto is 0 degree.

The pitch angle is a slope angle θ in the vertical direction formingwith two image pick-up devices 101, as shown in FIG. 9. Here, the yawangle is a tilt in the horizontal direction, that is, the yaw angle is 0degree if the road is straight. The roll angle is a tilt about anoptical axis, that is, the roll angle is 0 degree with the horizontalplane.

The terms “slope angle” and “slope degree” are different, and a slopedegree changes with respect to a slope angle. Details will be describedlater.

3-1. Theory of Slope Degree Calculation Section 13

According to Affine GP constraint in the above Patent Literatures 1 and2 (H. Hattori and A. Maki, Stereo without Depth Search and MetricCalibration, in proc. of IEEE Computer Society Conference on ComputerVision and Pattern Recognition, CVPR 2000, pp. 177–184, 2000), about 3Dcoordinates of a point on the road plane when a point of origin is amidpoint between two cameras, the equation 7 representing therelationship of corresponding points between the stereoscopic images canbe expressed much simpler. This is applicable with an assumption thatcoordinate values in the optical axis direction of the cameras aresufficiently large for coordinate values of the image in vertical andhorizontal directions (z>>x, z>>y, in which z denotes the optical axisdirection, x denotes the horizontal direction of the image, and ydenotes the vertical direction of the image).

That is, assuming that a point as a result of projecting a certain pointon the road plane onto the right image is P(u, v), and a pointcorrelating thereto in the left image is P′(u′, v′), the relationship isexpressed by the following equation 10.

$\begin{matrix}{\begin{pmatrix}u^{\prime} \\v^{\prime}\end{pmatrix} = {{A\begin{pmatrix}u \\v\end{pmatrix}} + \begin{pmatrix}t_{u} \\t_{v}\end{pmatrix}}} & (10)\end{matrix}$Herein, A denotes a 2×2 affine transformation matrix, and (t_(u),t_(v))^(T) denotes a vector indicating translation. To derive those,three or more pairs of corresponding points on the road plane are usedin a similar manner for h in the equation 6. By deriving those inadvance before obstacle detection, as long as the relationship betweenthe cameras in position or posture remains the same, there is no need toderive those twice for successive obstacle detection.

Further, if both cameras are oriented toward a vanishing point of theroad, and if rotation about the optical axis of the camera is smallenough, A can be approximated by the following equation 11. The affinetransformation matrix A will be dependent only on λ

$\begin{matrix}{A = \begin{pmatrix}1 & \lambda \\0 & 1\end{pmatrix}} & (11)\end{matrix}$

The positional displacement of the corresponding points in thehorizontal direction in the right and left images is called a parallaxd, and assuming that d=u′−u, the parallax d is dependent only on theimage coordinate v. It can be simply expressed as the followingexpression.d=λu+t _(v)  (12)

Considered now is a method using the relationship for locally estimatingλ.

First, a local detection region is determined, and a reference point isset to o. From coordinates V_(o) of this point, a displacement v−v_(o)is newly set as v. Assuming that the detection region is a plane or apart of a plane, and λ and v in the detection region are displaced by Δλand Δv, respectively. The resulting parallax caused thereby is newly setas d, and the following equation is established.d=Δλv+Δd  (13)

The first term on the right side of the equation 13 Δλ×v is dependent onv, and the second term Δd is a parallax element generated by Δv but notdependent on v, being constant in the local region.

With two or more pairs of corresponding points P_(i)(u_(i), v_(i)),P_(i)′(u_(i)′, v_(i)′) (i=1 . . . N) at hand, the fluctuation value Δλin the detection region of λ can be derived. In detail, with unknown Δλand Δv, the parallax d_(i)=u_(i)′−u_(i) and the resulting value v_(i)are substituted into the equation 13, and the simultaneous linearequations as a result thereof is solved. Thus derived fluctuation valueΔλ is the “slope degree” as will be described later.

If the feature points are all on the road plane, coordinates of pairs ofcorresponding points are supposed to satisfy the equation 10, and thefluctuation value Δλ is roughly equal to 0. On the other hand, if thefeature points are not on the road plane but on any obstacle, theequation 10 is not satisfied, and |Δλ|>>0.

As such, the fluctuation value Δλ is a factor for determining whether ornot the detection region include any obstacle therein.

3-2. Description by Way of Illustration

Referring to schematic drawings, the principle of the slope degreecalculation section 13 is described.

3-2-1. Calculation of Fluctuation Value Δλ

FIG. 6 is a schematic diagram showing a case where image regions R andR′ are previously set in the right and left images as both detectionregions. Both detection regions each include white lines and a roadsign, but no obstacle.

In the right image region R, six feature points P₁ to P₆ are found, andfor each thereof, a corresponding point is derived in the left imageregion R′, points P₁′ to P₆′.

Similarly, FIG. 7 shows an exemplary result of finding correspondingpoints in the region R′ with respect to feature points in the region Rincluding an obstacle.

First, in FIG. 6, focus on a line segment P₂P₅ in the region R and thecorresponding line segment P₂′P₅′ in the region R′. FIG. 8 showsenlarged version of these line segments as s and s₁′, s corresponding toP₂P₅, and s₁′ to P₂′P₅. The line segment s′ in the left region R′ is aresult of converting the line segment s by the affine transformationequation 10 based on the approximation of the equation 11. The reasonwhy the line segment s′ in the image has a different tilt from the linesegment s is that |λ| is not 0 but large. Due to disturbance caused bydisplacement of the reference point o or vanishing point, the linesegment s actually corresponds to a line segment s₁′ locating at aposition displaced therefrom by a constant Δd. Thus, solving theequation 13 derived from the coordinates of the feature points and theircorresponding point, i.e., P₂, P₅, P₂′, and P₅′, will lead to asolution, Δλ being roughly equal to 0.

Next, in FIG. 7, focus on a line segment P₁P₃ in the region R and thecorresponding line segment P₁′P₃′ in the region R′. FIG. 8 showsenlarged version of these line segments as s and s₂′. In this case, theline segment s is not located on the road plane, and thus the linesegment s₂′ corresponding thereto has a different angle from the linesegment s′ derived by subjecting the line segment s to affinetransformation. That is, to transform the line segment s to s₂′, thetransformation should be carried out in such a manner as to laterallydisplace points on the line segment by an amount proportional to v. Andsolving the equation 13 using the coordinates of P₁, P₃, P₁′, and P₃′will lead to Δλ, satisfying |Δλ|>>0. In this example, assuming that theprotuberance and hollow of the backside of the obstacle (vehicle ahead)are sufficiently smaller than the distance from the cameras, and anobstacle surface is regarded as a plane (i.e., detection plane) andapproximately perpendicular to the optical axes of the cameras. Undersuch assumptions, the line segments s and s₂′ become roughly parallel toeach other on the original image, thereby rendering Δλ roughly equal to−λ (at this time, A is a unit matrix).

Described above is a case of calculating Δλ using a line segment betweentwo corresponding points on the detection plane in the detection region.

3-2-2. Calculation of Slope Angle from Basic Plane of Object Plane

The affine transformation parameter is derived utilizing a fact that theparallax d in the equation 13 is dependent only on a v coordinate of afeature point. This is equivalent of deriving a pitch angle (i.e., slopeangle) from a road plane (i.e., basic plane) in the detection region, inwhich the detection region is a plane in 3D space or a part thereof(this is the detection plane). Herein, as to the detection plane, yawand roll angles other than the pitch angle are presumed as beingsufficiently small.

Referring now to FIGS. 9 and 10, this is described.

The slope angle of the detection plane corresponds to θ in FIG. 9.Approximation such as the equation 11 is applicable to a case where onlythe slope angle θ changes. If this is the case, in FIG. 10, the equation10 means that the rectangular detection region in the right image istransformed into a parallelogram in the left image. Displacementoccurring at the time of such a transformation is dependent, directly,on the size of λ+Δλ, and the actual cause is the size of θ. As shown inFIG. 9, when there is a road sign on the slope ahead of the road, andwhen the detection region is around the road sign, the detection regionis transformed into a parallelogram of intermediate state consideringroad being horizontal and obstacle being vertical. Depending on the sizeof θ, Δλ will take an intermediate value between 0 and −λ. Accordingly,by solving the equation 13, derived is the value Δλ showing a monotonouschange depending on the size of the slope angle θ.

Further, in the above example, the rectangle is transformed into aparallelogram using approximation of the equation 11. Accordingly, thefeature points and corresponding points do not necessarily aligned onthe linear lines, but only need to be located on some plane.

Therefore, in FIG. 6 example, by solving the equation 13 derived fromthe pairs of feature points and corresponding points, P₁, . . . P₆, P₁′,. . . P₆′, Δλ can be derived with higher accuracy compared with a caseof using a pair of line segments. Even if the feature points andcorresponding points are found on an obstacle having protuberances andhollows, such an obstacle is regarded as a plane if the distance fromthe camera is large. Accordingly, using every feature point andcorresponding point, a slope angle of the detection plane (obstacleplane in FIG. 10) can be approximately derived.

As such, Δλ is a quantity derived depending on the slope angle θ of thedetection plane, and Δλ can be forwarded to the result determinationsection 14 in the next stage as the slope degree.

Here, as the slope degree, a unique value derivable from Δλ (e.g., |Δλ|)will do.

3-2-3. Other Calculation Method for Δλ

Generally, when the detection region includes m feature points, and foreach thereof, n corresponding point at the maximum, m×n pairs ofcorresponding points can be derived at the maximum. Solving the equation13 derived from all of these pairs will lead to Δλ. The equation can besolved using two pairs of corresponding points, but three or more pairswill make the equation redundant. Thus, a statistical technique such asleast square is effective to derive Δλ with higher accuracy.

Further, using a technique as Hough transform eliminates the need forsolving the equation to derive Δλ.

This is described by referring to FIGS. 11 and 12.

Solving the equation 13 is equivalent of deriving an intersection pointof lines derived from a plurality of pairs of corresponding points.Herein, the equation 13 is regarded as a linear line in Δd−Δλ space.

As shown in FIG. 12, the Δd−Δλ space is divided into a plurality ofsquare cells, and when a line passes though the cells, the correspondingcells are increased in value. Such a process is referred to as votingprocess. After subjecting such a voting process to every possible line,finding a cell position showing the largest value leads to Δλ. Such amethod is effective even if not all the lines intersect at a point, andif a plurality of maximum values are found, the cell position showingthe largest value leads to Δλ, for example.

Similarly, referring to the right side of FIG. 12, after the votingprocess, by calculating a profile through an addition of cell valuesonly those being larger than a certain value in the lateral direction,the distribution of a can be known. The distribution result may beforwarded to the next result determination section 14 as thedistribution of the slope degrees.

3-3. Case Where Detection Region is Extending Over Obstacle and RoadPlane

Considered now is a case where, as shown in FIG. 13, a detection regionis extending over an obstacle and a road plane, and feature points andcorresponding points are detected both on the obstacle (P₁, P₃, P₆, P₁′,P₃′, and P₆′) and the road plane (P₂, P₄, P₂′, and P₄′), or detection ofsome corresponding points have gone wrong (P₅, and P₅′).

Assuming that linear lines drawn from the pairs of P₁ . . . P₆, and P₁′. . . P₆′ to Δλ−Δd space are lines 1 to 6 in FIG. 14, the resultingintersection point will be located near Δλ=−λ. This intersection pointis distinguishable from others derived from pairs of correspondingpoints on other road planes or pairs of wrong corresponding points.

Further, if the pairs of corresponding points on the obstacle aredominant in number, the voting value of the intersection point afterHough transformation becomes high.

Accordingly, with such a criteria as an intersection being closer toΔλ=−λ, or the voting value thereof being large, only the pairs ofcorresponding points locating on the obstacle can be distinguished. WithFIG. 13 example, only the points of P₁, P₃, P₆, P₁′, P₃′, and P₆′ can bedetected through distinction as belonging to the obstacle.

4. Result Determination Section 14

The result determination section 14 refers to the slope degree of thedetection region or a distribution thereof derived by the slope degreedetection section 13, and determines whether or not the detection regionincludes any obstacle.

Δλ calculated by the slope degree calculation section 13 shows amonotonous change depending on the slope angle of the detection region.When the plane in the detection region is almost parallel to the basicplane, Δλ is roughly equal to 0, and when perpendicular, Δλ is roughlyequal to −λ.

Thus, in the simplest manner, a threshold value is so set in advance asto accept the FIG. 9 slope including no obstacle, and through comparisonbetween Δλ and the threshold value, a determination can be made whetherthere is any obstacle.

For example, as to a certain threshold value th, if |Δλ|>th issatisfied, it is determined as there being an obstacle.

Moreover, if the slope degree is at hand as distribution, a statisticaldetermination using the distribution may be used to make such adetermination.

For example, referring to the right side of FIG. 12, the area above this p₁, and the area below th is p₂, it is determined as there being anobstacle when p₁>p₂ is satisfied.

Viewing the distribution as probability distribution, a posterioriprobability p of including an obstacle can be derived by p₁/(p₁+p₂), forexample, and when p>0.5 is satisfied, it may be determined as therebeing an obstacle.

As described in the foregoing, using a slope degree of a detectionregion derived by the slope degree calculation section 13 or adistribution thereof, by an arbitrary method, a determination can bemade whether there is any obstacle.

As such, using a plurality of cameras mounted in a vehicle or providedabove a road, even if the relationship between the road plane and thecameras constantly changes in relative position or posture due to cameravibration or a change in road tilt, any obstacles located on the roadsuch as other vehicles ahead, parked vehicles, and pedestrians on theroad can be detected without confusing those with textures includingwhite lines, road signs, paint, road stains, and shadows of roadsideobjects, all of which do not disturb vehicle driving. Further,unnecessary operations such as erroneous warning or unnecessary vehiclecontrol can be reduced to a great degree.

MODIFICATION EXAMPLE 1

Here, the present invention is not limited to the above embodiment, andit is understood that numerous other modifications and variations can bedevised without departing from the scope of the invention.

For example, the detection region is a square in the embodiment, but theshape is not restrictive thereto, and any predetermined arbitrary shapewill do.

MODIFICATION EXAMPLE 2

In the above, the detection regions R and R′ are predetermined in theright and left camera images. Alternatively, such an obstacle detectionmethod as described in Patent Literature 3 (JP-A-2001-154569) may beused for a preprocessing, and a result derived thereby may be used toset detection regions. To set detection regions, some driving lanedetection method may be used, and a result derived thereby may be usedunder a predetermined method.

MODIFICATION EXAMPLE 3

Even if no detection region is set in advance, the right and left cameraimages may be scanned, and at each scanning position, the operation ofthe embodiment may be executed for a similar object detection process.

MODIFICATION EXAMPLE 4

In the present embodiment, the equation 10 and other between-imagerelational expressions and drawings are provided for conversion from theright image region to the left image region. Those are not surelyrestrictive, and all allow conversion from the left image to the rightimage.

MODIFICATION EXAMPLE 5

Described above is a case where two of the image pickup devices such ascameras are provided. The number of the devices is not restrictive, andeven if three or more of the image pick-up devices are provided, theembodiment is applicable to a combination of arbitrary two image pick-updevices. If this is the case, the results derived from every possiblecombination maybe integrated together to realize the object detectiondevice with a higher accuracy.

MODIFICATION EXAMPLE 6

The road plane is assumed as being plane. The present embodiment isapplicable, even if the road surface is curved, it may be partiallydivided to approximate it as the plane.

MODIFICATION EXAMPLE 7

The object of the present invention is not limited for helping safedriving and realizing automatic driving of the vehicle, or for countingthe number of passing vehicles on the road or monitoring those passingvehicles for their driving. Providing the cameras to the rear part ofthe vehicle allows rear monitoring, and to airplanes or helicoptersallows object detection at the time of takeoff and landing. Also,providing the cameras to industrial or household robots allows objectdetection or monitoring for automatic driving. As such, the presentinvention is applicable to various many applications.

INDUSTRIAL APPLICABILITY

As is known from the above, according to the present invention, nomatter what road an obstacle detection device is set up, or no matterwhat road a vehicle having the device mounted therein is driving, objectdetection can be achieved with high accuracy regardless of vibrationduring the device operation or vehicle driving. Specifically, thepresent invention successfully prevents the conventional problems ofreducing the obstacle detection accuracy due to frequent erroneousdetection especially around the texture on the road plane such as whitelines, road signs, paint, road stains, shadows of roadside objects andvehicles, and the like. Accordingly, only true obstacles can becorrectly detected, practically benefiting a great deal of effects.

1. A vehicle in which at least two image pick-up devices each pick up animage of a preset common detection region, and which determines whetheran obstacle is present or not in the detection region from the stereoimages picked up using the image pick-up devices, the vehiclecomprising: image input means for receiving the image from each of theimage pick-up devices; correspondence detection means for deriving aplurality of pairs of corresponding points in the detection regions ofthe received two stereo images; slope degree calculation means forcalculating a slope degree corresponding to a slope angle between abasic plane which is parallel to optical axes of the image pick-updevices and a detection plane including the derived plurality ofcorresponding points in the stereo images; and result determinationmeans for determining there is an obstacle when the slope degree of thedetection plane is larger than a predetermined value.
 2. The vehicleaccording to claim 1, wherein: the slope degree calculation meansregards the slope angle as a pitch angle between the basic plane and thedetection plane, and a parameter of an affine transformation matrixindicating a relationship between the corresponding points in the imagesor a unique value derived from the parameter as a slope degree.
 3. Thevehicle according to claim 2, wherein: the slope degree calculationmeans calculates the slope degree by solving an equation based on aparallax and vertical positions of the plurality of pairs ofcorresponding points.
 4. The vehicle according to claim 1, wherein: theslope degree calculation means calculates the slope degree by votingcoordinate values of the plurality of pairs of corresponding points intoa voting space based on an equation satisfied by the coordinate values.5. The vehicle according to claim 1, wherein: the slope degreecalculation means detects, as an obstacle, out of the plurality of pairsof corresponding points, only the pair of corresponding points resultingin the slope degree large in value.
 6. A vehicle in which at least twoimage pick-up devices each pick up an image of a preset common detectionregion, and a determination is made from stereo images picked up usingthe image pick-up devices whether or not the detection region includesan obstacle, the vehicle comprising: an image input section forreceiving the stereo images from the image pick-up devices; acorrespondence detection section for deriving a plurality of pairs ofcorresponding points in the detection regions of the received stereoimages; a slope degree calculation section for calculating a slopedegree corresponding to a slope angle between a basic plane which isparallel to optical axes of the image pick-up devices and a detectionplane including the derived plurality of corresponding points in thestereo images; and a result determination section for determining thereis an obstacle when the slope degree of the detection plane is largerthan a predetermined value.